Description of the Change

Adding in a new Feature, Smart 404, that uses the power of embeddings, combined with vector search capabilities in Elasticsearch, to recommend content when users land on a 404 page, with the hope of recommending the actual item they were hoping to reach.

As mentioned, this Feature requires two services in order to function:

1. A Provider that can generate embeddings (right now we support OpenAI and Azure OpenAI)
2. A data storage engine that can also run vector queries (currently only support Elasticsearch powered by ElasticPress)

Once both are in place, all of your existing content (and any new content added in the future) is automatically sent off to the embeddings Provider. This vector information is then stored in elasticsearch and we can then run vector queries on this data. In particular, we've introduced new helper methods that can be dropped into a site's 404 template that will render these results.

At a high level, here's how the process works:

1. When an item is saved, the content is broken down into smaller chunks. Each chunk is then sent off to an embeddings Provider (like OpenAI)
2. The returned vector data is synced over to elasticsearch using ElasticPress
3. When a user lands on a 404 page, we pull the last slug out of the URL. For instance, if someone is trying to go to example.com/news/interesting-news-article, we take the slug interesting-news-article
4. This slug is then sent off to the embeddings Provider and the returned vector data is used to make a vector query against elasticsearch
5. There are multiple configuration options here that can be used to modify this query but we eventually end up with results back which are then rendered to the user

Closes #794

How to test the Change

1. Follow the instructions in the README to get things set up. This will include setting up an embeddings Provider and ElasticPress + Elasticsearch
2. Run an ElasticPress sync from scratch. This can be done in the admin or using the WP-CLI command that ElasticPress provides. Important to run this from scratch so the proper mapping is sent
3. Utilize one of the new helper functions to output results on your 404 template. Ensure this renders properly and that all various settings work as expected

Note it's helpful to install Query Monitor and the ElasticPress Debugging Add-On to see the actual elasticsearch queries and results

Changelog Entry

Added - New Feature, Smart 404, which provides the ability to render recommended content on a 404 page

Credits

Props @dkotter, @iamdharmesh, @berkod

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added Critical Flows, Test Cases, and/or End-to-End Tests to cover my change.
• All new and existing tests pass.